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Two papers on teaching methods are presented. The 
first concerns an experiment that tested the relative effectiveness 
of three teaching methods: an uninterrupted lecture/. a 
tape-recording , and reading a prescribed text. The three-way research 
design used the three teaching methods, three groups of six student 
teachers of physical medicine between the ages cf 2*1 and 42, and 
three psychological topics as sub ject matter . ^aqh teaching method 
was followed at once by a multiple choice test designed to measure 
eight cognitive levels froni Bloom's "Cognitive Taxdnomy of 
Educational Objectives." These levels of thinking^ are : terminology, 
facts, generalizations, understanding, application, analysis, 
synthesis, ,and evaluation. A. discussion of these concepts, testing 
conditions, and results of the pilot experiment is provided. The 
second baper concerns objective tests of the effectiveness of 
teaching methods in higher education. The correspondence between 
logical processes in the student's mind and categories of Bloom's 
"Cognitive Taxonomy of Educational Objectives" are expressed in terms 
of propcsitidnal, or truth functional, logic. The analysis concerns 
testing: a student's knowledge of fact presented during instruction, 
knowledge of presented relations :betweeen facts, unpresented 
relations between presented facts, the ability to apply presented 
principles and generalizations, s the ability to analyze unstated 
generalizations, unpresented relations between presented facts, the 
ability to evaluate, and value judgments. (SW) 
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its -terminology to 1^ ™?^'*.?'°™ knowled 9e of 
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The procedure was such that for each topic (S, G, M) 
a lecture was given to one group 6f students without - 
interruption and lasted for 20 to 30 minuses. < The 
lectures were recorded and playetd to a second group 
of students without the teacher being present. "Care 
was taken that where the lecturer (L) had used the 
blackboard either for illustration* or for emphasis 
of certain words, the same stimuli/ were available in 
condition T. , - 

A typist duplicated the lecturea using, a dictaphone 
machine. Punctuation was at her discretion. s 
Fundamental illustrations were also duplicated,- and - 
words that were written on the blackboard were 
underlined. 

Each teaching method (L, T, R) was followed at once 
(allowing time to give out question and answer papers) 
by a multiple choice test at the following eight 
<j?ognitive levels': * ' 



Table 1. ^ 



Level 


1.1 Terminology 

"X 


- . — 11 1 

ie recognition of a naiue or term ^ » 


ii 


1.2 Facts 


, : * . \ ~- 

ie recognition or recall of specific facts 




1.3 Generalisations and Principles 


ii 


2.0 Understanding 


ie simple interpretation and relations of facts 


n 


3.0 Application 


. ie facts and principles 1 




4.0 Analysis 


ie of data given in the question or in S,G or M 
"""" 


ii 


5.0 Synthesis 


including- a plan proposal o* creative act 




6.0 Evaluation 


judgements about facts in fi, G or M 



The students who were taking a fairly intensive course in 
Phenology for two weeks before this experiment, were 
?elat?vely "naive" psychologists. "/Hence, although^ one 
relatively nai U seems reasonable to suppose that the 
thJnkLg rl^uirea'to anlwer the higher level questions was 
largely donl while .receiving the information or during the 
tests, and not before. This point is important if^one is 
to assert that the differences in cognitive level are . 
attributable to the' methods of receiving, information. To 
avoid the criticism that the thinking at higher levels may 
have been stimulated by questions earlier in th e ^st, 
questions at different levels were placed in a random order. 



^L^^g^e^r^^^ fWchoice 
in the setting o que t ^ W tW ° cont rols 

to observe, but which m v L! h are ver * difficult 
(a) devisiAg dis£rac£ors Y w^h* COmm ° n sol utipn:« , 
probability\i t rSe?s in ?h not ° nl Y °f equal 

comparablewith thos? in ot^ same question, but are 
that the differences in d?J5? ?^ es J i °ns in such a way 
of different cSqni?Le if 1 f ° Ulty bet ween questions • 
differences in mental ninn! S derive d from 'the 

the fineness of Ss^iSnatfn^ 

its alternatives; ^b) const^n^ COrrect a *swer from 
particular coaninJ L f Cting questions at a • 
admits that (of £ liffltul^ S 1 ? 0 ?' 8 SCal€ > Blodm 
further complicated tt -~ ' ^ ifc seems & be 
concepts wKln B?oom' s PP ?M f sub - b ^archies of 
following groups of may be < Seen fr ™ the - 

HnstrumLLrL^ro?ing a : S ° Ciated Sk±nner ' s 



(A) Food, rat, lever, milk, 
(c) 



(B) Stimulus, ^response, motivation, reinforcement, 



^t^e^^^^ 0 ^' Shaping, extin^on 

' ^e rf wo C r^ 1 cou\^ W b ° e Ul S ^ th ? ™^ of 
but those in (M arp lGVel "Terminology" 

any teaching; indled he mu^ ^ J"?*" before ^ does 
Those in (B) are tlchnin^ - a ° h by * sin 9 them, 
in terms of (a7 or^Jse J ?"* C f^ hle of definition* 
Being technical tlrhl '^ZSLVFT 1 ?* described in (A).. 
Words at level ( * i in ?l a " 9,lt ^ the teacher, 
already defined by ££ Tel chlr ? ^ he ' tec ?nic f 1 terms 
the student must have moJe ?hfn level (B) '• Thus 

learn words at level (c) ^, comm °n knowledge to, 
(C) usually involve combining m ° re WOrds at ^1 
previous level ?oV the2 Se?in??^ eptS fr ° m the 
an understanding of ?hnL £ inition -. This will require 
at level 2. 0 g oh ll^l^^^^^*^* ^ 
experiment questions teItl2rJh- Ther f fore in this 
contained in (C) were r!«2 J meaning of-, terms 
difficulty asS n der=t a S?n n « d as °f the same level of 
Words at Lvef ^^^2?'^ ^ (B) ' ' 

b^adriasLk? U i?? S ^f 2*5 P robl em (b) above can ' 
subject' to br t Jug^\S%?;^ t fS' d + COnCept8 ° f 3 

s^d^^ ■ 
as the grSff j l ^™ gg C0 ^eotual ^ 

discrimination required 2f If S 9 *! 6 , of 
possible. This also Insures ^hat^hr f S / he f±neSfe 
through all the conHff!! the st udent goes 

to obLin the ?ig C hrans^r Pr ° CeSSeS analySed * B1 ^ m 



To son* extent Bloon, anticipates the 

ttoalysis and Synthesis), not only level 2.00. 

Sft^^ £<£3 SS^'-S-tS^S teaching 
Sods eacn^howe^insignificant variations when 
Total scores were taken into account, when, these 

0tl Kiown b rn ke ta b fe W ; £erac?io»»s&- 
itllll anHeis of sludenSs remained insignificant, 

methods at the simpler cognitive levels. 
' as should be predicted Figure 1 also, shows that 
stuSs sco/e higher ^-^"J^SfforoncoB 
i^eeen'tSafsSrerft S^Si^SinSive levels, 
wouirSccSr^y'chance less than 1 in 1,000 times. 

Comments . In view of ^^^^^^f^heir^ 6 " 
recordin gs mentioned at the beginning, th ^ e 
superiority was wholly unexpectedly the jxpj^ £ 

It might be "l^^^p^ tS need to coventrate 
learning in the classroom, P-"-"^ . to qrea ter 

in case essential facts were support, this 

self-discipline; b^^f/^^L^LSSriS^etention 
interpretation which would require superi^ ^ _ 

^roTl^ng^^^ Could it be that 

2.00) toiiowing f verbal information- is - . 

auditory presentation ot veroa that the 

superior to a written presentation, *" « ti ra ther 
presence of a lecturer is actually g^gg^*^ 
than st i?ul«ting? « so, ^cassett^libr^y ^ . 
s^ousS considered Sy -itege^ibrarian^. Such a 
sTrv?ce could be mailable during normal Jours it^ 
could save teachers time, suitable . used 

. e peri^enfa^ « 
b^th require further empirical verifications. 
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It will be noticed from the graph that there is a 
sharp drop in scores between levels 3.00 and 4.00 
• It is interesting that in their previous traihinq' 
in physical medicine these students were required 
to learn terminology, facts, how to deal with facts, 
. and how to understand and apply them; but they 

were not required to analyse or resynthesise them in 
new ways, still less to evaluate them in relation to 
other fields of knowledge. -(See table 1). The 
prediction that students from other backgrounds would 
show a more gradual decline in scores with cognitive 
level has subsequently been confirmed with student 
teachers. This suggests that cognitive skills can 
be taught. 

the rZ"^ 2! P^ ^t'wt that, provided 

the rule of thumb is adopted, multiple-choice tests • 
using Bloom's Taxonomy are a useful tool for measuring 
cognitive ability and future experimenters on ?ea^hinq 
W«? dS r^ d bS u "wise to nealgct the higher coan^?L 
levels • if these could reasona bly be included amongst 
the teaching objectives. Judging from the range of 
^ incorrect answers selected, the rule of thumb is a 
useful tool for equalising the power bf distractors 
(problem a) . 

Criticisms ^ v ^ 

(i) The multiple-choice tests were all given immediately 
following the teaching in order ,.to eliminate the 
effects of subsequent reading and discission. This is 
clearly artificial as one of the purposes of teachinq 
in further education is to stimulate discussion and ' 
interest that will lead to further study. it is also ' 
a dubious measure of the effectiveness of the teachinq 
t in terms of "permanent" learning and in terms of Bloom's 
^levels.. Immediate testing, obviously favours retention 
and understanding of facts and principles (i.e. levels 
up to- 2.00). Analysis. (4.00), Synthesis (5.00) "and 
Evaluation (6.00). take more time. One learns to absorb 
the (facts during a lesson and then go away and think 
about them.- it may be, however, that: one would expect . 
level 3.00 (application) to be thoughtTabout after *a 
lesson not during it Yet performance at this level v 
is virtually as good.as the, lower levels. This 

S^itf 1 ??* 18 ^ inhe .^«t problem in doing any assessment 
• of „ the. effectiveness of teaching methods. Without^ 
testing ve-ry soon after teaching," it is very difficult ' 
to allow for the effects of subsequent activity. 

(ii) No generalisations should be made as this pilot 
te|;t was made using only one teacher. 

Uii) No measure of personality or intelligence Were " 
taken, but it is usually thought these are important 
in teaching and learning. 
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(iv) . It ccuid ie ^j^^*t?£r.£SL 

is itself an analytical si tuat^on^ ^ 
is required to select one ± an ability at 

Thus the nature of the test r ^ -led that most 
level 4 00. o f«^«'^™; y lr e some ability at level 
other forms of testing requii formulating a set of 
5.00 because essay writing or f °™ u ^ n n 9 a svn thetic 

words for the P^P°^ es °^ C °^ nl ?oo' ?s In inherent 
task.. Thus it may be that this, too, is 
difficulty in any test situation. 

(v) it is sometimes said that ^ffifi?^*" 
Ire not objective because there is a ^^ective 

fTeeVow this S c tm °e iMnatefand it seems fair 
To supp-Itnat ven if. mul ^P^-^^ob^c^ve^han 
not objective, they ,are at x east more The use of- 

^^^^^^ -ance in • 
overcoming this objection. 

Conclusions . It must be "T^SiSiif^SiSiS: ' 
pilot experiment and" that the following 
are therefore only provisional. 

(1) Bloom- ^classification of educational e Objectives 

' eSectivenesf of teaching methods Shen modified by the 
rule 6f thumb. 

(2 , investigations on ^ effectiveness^ teaching 
reve?fif h rhese 1 cou?frrason r ariy°oe h included 9 amongst. 
the teaching objectives. 

(3 , with reservations "seems possible that teaching 
by use of "corded lectures resul *s « described - 
. ^s^but'theret no C £ffete»ce at simpler cognitive 
levels. ' 

(4) This line of research looks as if it could be 
promising. 

r 
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FIGURE 1 : PERCENTAGE OF* CORRECT ANSWERS FOLLOWING LECTURE 
TAPE OR READING AT VARIOUS LEVELS OP DIFFICULTY ■ 
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GROPINGS FOR A DESIGN OF OBJECTIVE TESTS OF THE 
EFFECTIVENESS OF TEACHING METHODS / ; . ( ' 

There are two common criticisms of objective tests: 
' (1) they can only test factual knowledge, (2) they 
are only objective in the marking, not in the setting 
(Head 1968). The errbneousness of the first criticism 
has become more clear since the publication of sample 
questions, in Bloom' s -"Taxonomy of Educational ObDectives 
and a number of workers have used multiple choice 
questions at various "cognitive levels to test the 
effectiveness of teaching methods in higher education 
(e.g. Hoare 1970, and Bligh 1970). : v ; 

The purpose of this paper is to present some gropings 
towards an answer to the second criticism while 
maintaining questions requiring a variety of cognitive 
processes to avoid the first. It Is possible that the ^ 
style of test suggested could be used to test students 
value judgements, but this requires further investigation 
These gropings are only relevant to tests of the 
effectiveness of teaching methods administered after a 
period of tuition in which the statement s given to the 
student are specifiable. These statements^ will be 
called "propositions (G) " or " statements (G) ". 

When propositions (G) are specified it is possible to 
produce further. propositions, which may be called , 
question statements , statements (Q) or propositions (Q) , 
and which bear a precise logical relationship to them. 
This logical relationship is objective. It is open to 
public scrutiny and public verification according to 
the canons of logic which are also public. 
Propositions (Q) may be true or false and their . 
relationship to propositions (G) may be valid or invalid. • 
They are questions in that, although expressed in the 
indicative, a. student is required to say whether they 
are TRUE, FALSE or he DOES NOT KNOW. Therefore it is . 
possible to design a situation in which a student will 
answer correctly, if he makes certain logical, inferences 
which are specifiable. The converse does not follow. 
He may answer correctly without thinking logically 
either consciously or unconsciously. Nevertheless 
the probability of his doing so is describable according 
to statistical "laws" which are equally public, and 
hence, equally objective. The provision o| the "don^t 
know" category may reduce the proportion of guesses , 
and the incentive to guess may be further reduced by 
arrangement of the marking scheme (Crow Goldsmith and 
Diament 1969) . . 
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Since there is an infinite number of possible 
relationships between propositio/is «3) and propositions (Q) 
there is a task' of selecting particular relationships for 
the purpose of testing the effectiveness of "teaching. . - 
The relationships selected should correspond to the 
objectives of the teaching. Since this paper is 
considering objective test.s of the effectiveness of * 
teaching methods in general, and since -Bloom 1 s is a 
general statement of objectives , this 'paper is conc6tried ° 
with logical processes in the mind of ^he.. student that 
correspond to eight of Bloom' is categories" as v set but 
below. Since it is proposed' that students should judge 
the truth or falsity of propositions and the validity of 
relations between any propositions it will be most 
» convenient to consider these relations in terms of 
prppositionai, or truth functional/ logic. Propositions 
unexpressed during teaching and propositions (U) . " 



Bloom's Categories ^rue Propositions (Q) 



1.1 Knowledge of 
Terminology 


- - f (a(G)) . ■ 


1.12 Knowledge of 

Specific Facts 


(fa) (G) ' /;>^^ : 'Y : 


1.3 Knowledge of 

Generalisations 
and Principles ~ 


J4 / (ga o fa) (G) 


2, Comprehension 


r (U) 

where: p(G) . q(G) r> r ' V; 


3. Application 


-(g^o ;fa v ) (U)^ • . - : •-■/ 
.where a 1 is d "specific instance ; 
of 1 a 1 and » ' ga o fa ' Is given. 


4. Analysis 


; ; (a) (fa o ga) .(U) 
* or (a} (faa ^ ga) (U) ~ 
- or .rO* (a ) ifa o nj ga). (U) 
or u (a) (fa o ga) (U) 


5. Synthesis 

■V- 


v(U) 

where (p (G) • ,q (G) O r (U) ) 
. (s(G) ."tft(G) O u(U)) 
. (r . u 0 v) / 


6. Evaluation 

\ " 


r 1 (U) or (s l o r 1 ) (U) t: ' 
where (s D r) (U) . (p(G) . q (G)D s 1 ) 
and where .'r 1 V is an instance of 'r' 
and 's r ' is and instance of 's 1 
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■j^he presentation of Table I may lead to two 
misun&ter standings." There is no suggestion that 
the correspondence between the logical formulate 
and Bloom's levels is 1 : .1. Bloom's 
categories are broader and more blurred at the 
edges because they are intended to be , < 
. '„ - comprehensive aha because they have been formed 
by a concensus of subjective opinion. The • 
logical "formulae represent am attempt at precision 
' t and objectivity; and are hence restricted. 

Secondly although there is increasingly logical- 
complexity it by "no means follows that there is 
• - increasingly psychological difficulty. An item 
may be logically more complex if it has more 
parts; it Is at least arguable that some thought 
1 ■ . processes operate with whple£. (This misunder- 

standing. may arise thrcg^$&ng the word simple 
instead of "easy" as -tffeop^Ssite of "difficult . 
There is also, a confusion between the logical 
description ojfc thought" ^processes aTria. their logical 
validity.), $he degree bf difficulty of the 
questions is an empirical, not a logical, issue 
and is measured by the .^rnnpi; 

' ... _ Total niimber of correct answers 

Degree' of difficulty MaximunKpossible number ot 

I .-' • correct answers 

The complexity of thought processes is< one factor 
- affecting their difficulty. There are many others 
<L,» such as«the familiarity of concepts and what they . 
ff are concepts of. • 

TESTING TERMINOLOGY * - » 

in the symbolism of , the predicate calculus we ; >may 
write a simple statement or proposition 
(p, q.,*r, s, etc) as "fa" where "a","b", "c; 
8» ' ' or "a 1 ", "a 1 ", "a 5 " ...."a*") stand for arbltary. 
proper names, and *fV»g","h" (or "f 1 ", "f 
»»f 3 " . .. "f n ") stand for arbitary predicates, 
such that "fa" means "'a' has the property 'f 
Irt other words the predicate 'f describes or 
qualifies the subject 'a'. 



A proposition "fa" has the simplest form there can 
be f (with the exception, of existential propositions 
with which we are not^concerned for the purpose of 
testing) . This produces a -problem where we wish 
to test whether we have taught the general meaning 
of a proper name ('a'). The^proposition fa will 
a;ways be a specific use of f\. Inevitably fa 
tests the student's knowledge of the truth or 
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judged "True" or "False" t« «Zk ? fc is 

• tesled beccmes^nc^elge-of 1 ; fact (aSeft"^ £ ► 

definition which is a fact about it. cf . 'w^tgenstein) . 

It may be asked, "what is the criterion of generality' 
is this not subjective? If there is only a matter of 
' what ' is ?„r*E te = tlng *f™inolo g y and testing factsf 
them'" M b K ^ e criterion by which to distinguish 
general!^ J? •^? ptlng u to re Ply that the degree of 

2=*i assess ^a-'s-ri^^ 

';\" f nlb%r^1 i 1 f !| ren e i ( a a Lii%ie a snir l name '~ 3 - 
"f" can be narrowed by specifying the abilities anrl 
accut^ne ITAtl^r*^ -itting^aiJs • 

"f n a" = (Intelligence^) is (to do with abilities (f") . 
It ceases to be plausible where "fa" is false but 
could become true if »f is made more genera? (L if ' 
Intelligence is a name for divergent Ibi?Ses" is 

falaifv ?h- ««. J i boundary between truth and 

^W^^H'^^S SsVof th 
p?e r digate rt Sr re ^ ° r ^P-ent^f "Ihe^ 

iubjecri^ Soi'ce! Sel6Cted f ° r br °^"i»9 is a • 

ma."ifS*J ^ertous'onffn^thl u's^of " 

ii^iter^'s.' st4thl - ^ " - • 
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TESTING KNOWLEDGE ON PRESENTED FACTS 

To test a student's knowledge of a fafct presented 
during instruction, he is re-presented with the 
statement. At its simplestjbhis is avrecognition 
situation, but he may judgeTit true oifalse as a 
result of any number of more complex processes • 
using other information. To some- extent the \ 
possession of information dan be tested by a 
pre-test (although. this has its problems) , but not 
entirely. For example a student may not know 
"fa" in a pre-test ^'because he did not, know the 
term "a". In the post-test he may know "a" but 
not "fa", yet infer it from "ga" also given during . . 
teaching. To some extent, too, processes alternative 
to recognition may be excluded by choosing fa 
from propositions (G) so that there are^ f ew related 
propositions of the. kind, "ga" and "fb^ , ^ 
particularly if "gb" is also explicit. ^Sincre ga 
and "fb'Vhave an objective relation to "-fa", this . 
preserves objectivity in principle, but within the 
.ordinary language of 1 a teaching situation, sentence 
structures are complex and hence the .questions are 
difficult to devise. Furthermore if the teacher 
judges "fa" to be important and therefore worth 
testing as one of the cognitive objectives, it is 
most unlikely that this statement will stand in 
logical isolation amongst all propositions (G) . 
Indeed the relatedness of a proposition may well 
be a measure of its importance -/but that is 
another question. 

A further problem arises when testing this, and 
all objectives, that propositions (Q) where the 
correct answer is "false" involve a more complex 
thought process than if the correct answer is 
"true". A student- cannot "recognise" 3, 
proposition (G) as false (unless it was wrongly , 
taught -or was explicitly taught as being false). 
The realisation that a proposition is false 
requires some analysis of its meaning, which . 
recognition of whether It has been given before , 
.does not. This is particularly .true wheto 
propositions (Q) are in a list in which mostof 
the questions do not pose a recognition problem, 
so that the situation facing the student is not 
a simple question "Do I, or do' I not, recognise 
this proposition?". In the case of testing 
factual knowledge this problem cannot be solved 
by changing the nature of false propositions (Q) . 
It can be partially overcome by ceasing to insist 
that the testing pf factual knowledge should be 
'a 'precise reqogni.tion situation. For example, 
certain words may be omitted from a proposition (G) 



so that the resultant proposition (Q) may be either true 
or false while the degree of change from proposition (G) 
does aot vary according to its truth, value; - variation 
is the source of this . problem. . Because of these 
changes the student is now required to -remember the 
meaning of proposition (G) , not their form of words. 
More precisely/ it is statements(G) that are to be 
. tested, not propositions (G) , where '"proposition" is 
taken to mean a form of words, and "statement" what 
the form the words says. When thinking in terms of 
Bloom's categories this change, is significant, because 
it ^introduces an element of "Translation" which^as 
category 2.10, is classified as testing "Comprehension", 
not testing facts. However the elements may be small 
and the fact that questions do not fit Bloom's 
^ categories does pot invalidate the test design^ • • 

Within the context' of this paper a more serious 
criticism .is that as soon as the test constructor 
• says "I have changed the wording, but I have not 
changed the meaning", tfiere is a strong suspicion of 
•subjectivity. A concensus of opinion on semantic 
rating scales (see Remmers " 1964) could relive this 
suspicion. , Alternatively the use of propositions and 
• their ~ negatives in paired pilot tests wouloS-show 
whether correct' and incorrect statements (Q) are of 
e^ual difficulty; and this is not open to the 
objection of confusing difficulty With complexity- 
because. variables affecting difficulty, apart from ' 
complexity, are held constant. k .'. . 

' TESTING^ KNOWLEDGE OF PRESENTED RELATIONS BETWE EN FACTS 
.-■ . : - '-. ', • : ' — ' J- : — ; — ; — - 

It!' is. normally an objective of teachers not only - to 
teach specific facts but certain 'relations between : 
them. The behavioural criterion -df '.this objective is 
that the*st;ud£ht should be able to, recognise statements 
of these^relaUons (not necessarily the precise •• 
propositions (G) . ) '• • y 

T 'J « 

In truth-functional logic the principal '.ways of 
relating .propositions are. by the" "logical 'connectives'"* 
of implication, conjunction, disjunction, and 
equivalence. For practical purposes implication , is 
the most satisfactory to test. To test conjunction 
is to do little more than test two facts in isolation. 
The ambiguity in the ordinary language in which the ' 
test is written, between the inclusive and exclusive 
disjunction could produce ambiguous questions. 
Statements (G) of equivalence are rare in teaching and 
consequently pose problems for the test constructor. 
Implication statements (G) are commpn and relevant 
because explanation is a very common objective of 
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teaching. There are many kinds of explanation t 
apVrt from logical implication, consequently the , 
relations between propositions to be tested 
should be broadened to include relations of cause 
and>f?ect, correlations and other implications 
accepted within the field of discourse. On the 
other hand for t£ie purposes of testing, these 
relations must be restricted' to those that are 
made explicit during teaching (ie they must be 
presented.) Many implications dm teaching are < 
contextual. These should not be included ; 
firstly because they are made by the student as 
a result of a thought process classifiable as 
"Comprehension"; secondly the inference is 
sometimes more intuitive than capable, of logical, 
expression,- and consequently the relation or 
statements (Q) to statements (G) is not objective. 

In order to ensure that, it : is the relation that \ 
is tested rather then the truth of the related . 
propositions, proposition (Q) may consist of an 
inversion of the explanadum and explanans. 
Emphasis oxi the implication, can be obtalned^if • 
proposition (Q) begins "The reason why ..... 

Testing presented relations between facts has no 
precise equivalent amongst Bloom's" categories. 
Bloom's "Knowledge of principles and generalisations 
is the nearest iti that principles are frequently 
expressed as implications and generalisations may - 
be; but the relations tested here may be quite 
specific and in this respect be closer to 
Bloom's 1.12 or 1.2,. . . \ 

TESTING UNPRESENTED RELATIONS BETWEEN PRESE NTED FACTS 

Not all the relations between facts that a teacher 
wishes the- student .to know, can be presented. The 
student must relate for himself. Consequently a. 
further teaching objective that should be tested is 
the student's ability to relate facts not explicitly 
related by propositions (G)^ This is a measure of a 
student's "understanding" (and "interpretation ). 

" if two propositi 6ns (G) in conjunction imply a third, 
this may be stated as a proposition (Q) under. this . 
headingVovided.it was not presented during teaching 
That is, if "P . q ^ r", "r." may be a proposition (Q) 
if it is also a proposition (U)Y False conclusions 
may be invented. t 

These questions' are not easy to construct because, 
the constructor must make sure .that "r" cannot £6 • . 
inferred from proposition (G) other than p . and q 



a correct answer (other than by chance) The mS? 
Inferences'^ not" r 6 * f ?Pec«ying thesf 110 ^ 9 

' might Se^t'first £2? ""^^ £ °™ rd <" " 

= theT g ro„ h oiSr^,^ CreaSe in 

= f "The'S???^ children dp not increase' so much" (G) 
. i fferen f e ln intelligence between normal 
oWer"(Q"?5T a 0hlldr " n lnoreas « they°g?o w 
"r^,is supposed to be a valid' inference frrtm a ^ 'A 

iyirl^ss^e h n I' H * = ^ 

proposit ion m? ZkII* » w q f either assumes^hlT 

proposition (U) that "sub-normal children are lesc, 
intelligent than normal children" or the ?b?lifv ^ 

^ferenM^^ 




aoo?^ e ^? bj ? C 5 ::LVe 18 that indents should be able to 
apply a principle and recognise particular instariiL « 

2h5 r 2S t SiS n :f ali " tlo ? ? Th ^ -y cu „or tTJrTill of 

the -3- 2k ? f prinol P le ">ay be expressed "if 

"a??""' is a ' f ?" ^ 3 Sl ? gle P r °P°^"°n of the form 16 
one instate n " here ( th f -^ject 'a' has more than 

issSe). lmport * nce here unless a guestio^begs thig 

so o m ; a .v s is f : f .- • ( ni?r°a> f n\ 

Some -a- is not -f-V ~ (a) (gf 3 ° Taf^ 

Thus generalisations may be expressed as implications'. 

J; 8 ?"^*' 8 a 5 ilit V / to apply a principle' or^Skise 
?n J£S?SS< 0f 3 ^realisation may be tested S^prlLntin 
an implication statement, (Q^in which the subject^l the 
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first proposition is an instance of ttfe principle 
or generalisation afid in which the second draXSs 
an. appropriate conclusion (or ina PP^ i ?5 e Bpa r se - ) 
conclusion,, if the correct answer. is to be False.). 
5Ss ' requires (first a realisation of the -Principle 
or generalisation to be applied, and second, the 
•• - .student's abifity to apply it (ie fo draw the 
S -conclusion), fin .practice the left-hfnA side of 
* <he implication may not need to be stated at all > 
(eg "Aritrxmetic^equires convergent ^f 1 ^^ 4^ 
Jhere, the gWal^ion is "Convergent thinking 
^£ required" ^here there, is^only one right answer 
^ . to a problem ") or -A W require two propositions 
(eg "If a child is 6 ytears"old and has an I.Q. of 
•120; his M. A. is 7.2"-)Y Yet if th*! form of the 
• . propositions (Q) • is variable can It te cl aimed^. 
' that- -the mental probers required of -the student 
is- always the same? this, strikes jt the basic, 
assumptions -behind thl claim of objectivity in the 
test Resign. ) 

TESTING THE ABILITY. TO^NALYSE UNSTAT ED GENERALISATIONS 

We have seen above that the four classical subject- 
predicate forms of generalisation may be expressed 
-as implications in truth-functional logic • 
*gtrawson 1952). Therefog if these ,5eneralisations - 
axe presented as ordinaryH.anguage statements (Q) , > 
■?hey Say be judged "True" or "False" by students L 
- The instances' that form the generalisations (whether 
truthfully or not) should be obtained from the 
teaching statements (G) . . '\ 

If a generalisation is expressed in the 'form 
(aHfa o ga), or one .of the other three forms given 
ibove and^n Table I ^the. student^has to recall 
each instance of " fa" 'and - check that it is also ga . 
^eg "Each of Binet ' s ^categories of intelligence has 
the same range of I vQ. " > "where the ranges in I.Q, 
were given in a number of proposition (G •) This is 
an analytical task because it includes identifying . 
a quality that is part of a concept. 

TESTING' UNPRESENTED RELATIONS BETWEE N' PRESENTED FACTS . 

As it stands this heading is inadequate -Clearly one 

^cannot be concerned with un P resen ^ d f | c f ^at it 

must be relevant, ^hey may include one fact that is 
iudged to be commo^JknJwledge or that the student 
should know from previous study, and one, or both, 
that may be logically derived from propositions (G) . , S 
« The logical formula by which they are derived need 
St ^invariant but' the paradigm is (p q or) . and 
it should be simple: The formula for this stage is 
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(r . u o v) where "r" and "u" are derived statements (u) i 
not statements (G) and "v" is classifiable as both (U) . 
and (Q) . • *'"»... 

This kind of question tests synthesis of what is 
"understood." *nd is analogous-td Bloom's level 5.00. 
It may be interpreted as testing .a synthesis of two"' 
statements testable at level 2.00; Factor analysis, of 
test scores should confirm this interpretation and may 
be used as a ^check on abuse of the variant derivation 
of the . synthesized statements.^ * 

Allowing this variable 'derivation' is a concession to 
-'subjectivity owing to -the difficulty of producing level 
2.00' statements (Q) previously mentioned, and which occurs 
three times over in~ the construction of questions at V 
thris level. "• . , ^ " ' , ^ 

< ■ - - * . 

" NESTING 1 THE'' ABILITY TO EVALUATE- ' i ' ' 

' m ' ' : — ; -r V 

Evaluation consists -of drawing a conclusion by using 
an .unstated principle (or generalisation) and a derived 
fact as an ^instance of that principle. °In other words 
it is assumed that the student already possesses 
certain principles or generalisations (eg "Introspective 
observations are subjective"). As when testing 
application the statement (Q) is normally an implication," 
but may appear as a simp-le proposition if reformed as 
a, generalisation "all 's' are •r , " f or if the left-hand 
^ide of the implication is low in information content 
or is otherwise redundant. In terms of Bloom's levels 
this kind of question involves a combination of 2.00 
and 3^.00.- That this is so is\ similarly confirma&'le 
by factor analysis of test responses. 

^ TESTING VALUE JUDGEMENTS , 

Provided they are expressed in the form of an 
implication or a subject-predicate proposition, there 
.is no reason why- both ndral and non-moral value 
judgements should not >e tested by this method. 
(Indeed it is possible that this method offers a -more 
valid measure where af£e%tive statements are hidden 
amongst cognitive ones, >than where students are 
concious that their values are being examined - but 
this is mere speculation. ) Moral judgements may be 
expressed in the form of an implication (eg -"If (the 
patient is suffering from shock) (p) then (he should 
be wrapped in a blanket )• (q) "). • Non-moral, judgements 
may be expressed by using a value, or "emotive", 
term in . In a subject-predicate proposition 
(eg "Massage is unhygienic"). 
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It is sometimes objected that value judgement 
are subjective and cannot be tested, but in the 
present context they are no more subjective than 
other logical thought processes. ^^V s tr . ue 
th^t there are additional logical rifled for value 
judgements,- but they do not prevent the use of 
this test design and will not be discussed here) • 

A second objection arises from the belief that 
''there could be ho objective Canon of "value 
knowledge" corresponding to the^verif ication, of 
empirical knowledge. But in vocational education 
there* are frequently explicit professional ethics 
J^which may b^tested by assent or denial to a 
'proposition expressed in public language. , . 
Furthermore this no objectipn within the 
* context "jpf testing teaching methods where this 
assent/constitutes the teaching -.objectives . 

' ■ -*v. 
'} VALIDATION OF THE TfiST DESIGN . 

Since claims hav<* been made,- for the objectivity of 
this test design one might expect that there should 
be some external criterion to validate these claims. 
It is possible that a student who does well at, say, 
application of principles will be good at applying 
other principles.* We might also expect that thi£ . 
student would be good at applying the same 
principles in different conditions, In other words , 
there is some validation if the skills tested are 
transferable to other subject matter and to oth«: 
conditions. If the skills do not transfer then 
the difference may be either in the subject matter 
or the conditions. In neither case singly is the 
test invalidated, for the difficulty of thfc patterns 

••■ of logi£ar* inference may vary with the subject 
• matter, and if they- vary with the conditions that 
is what the test is designed to show. If test 
result? vary with the same subject matter and the 
same conditions (this includes control fox 
individual differences) the tes<t may be invalidated 
because 'It is -unreliable. Reliability - is a measure 

- of consistency which is the essence of logic. It 
is not a coincidence that, it is only though this 
concept that the' test design can be invalidated and 
thatTit is an internal criterion. Empirical tests 
alon^ cannot invalidate logic. 

It might be thought that the logic could, be validated 
by reference to some external logic of language. 
If there was. such a logic (such as that proposed 
by the logical atomists) this could in principle be 
done, but Wittgenstein (1953) has 'shown that there 
can be no such logic^^JThis is why this test design 



is Only suitable for testing the relative ^effectiveness 
of teaching, method, the relative abilities of students 
or other variables within ' a given situation. It is a 
-relative measure, fro* external ' validation J.s possible 
with certainty; " ' / 

. • ✓ 

It is, however, possible to establish norms so long a& 
£he subject matter ,* categories of propositionsXQ) , test 
Conditions, thatching methods and groups of stud£*f£§^ 
are specifies.: The lack, oif certain external Validation 
„ is not a defect of the t£St design so long as it is not 
used for purposes for which it is not- interraefl. 
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